Adaptive learning algorithms to incorporate additional functional constraints into neural networks

In this paper, adaptive learning algorithms to obtain better generalization performance are proposed. We speci"cally designed cost terms for the additional functionality based on the "rstand second-order derivatives of neural activation at hidden layers. In the course of training, these additional cost functions penalize the input-to-output mapping sensitivity and highfrequency components in training data. A gradient-descent method results in hybrid learning rules to combine the error back-propagation, Hebbian rules, and the simple weight decay rules. However, additional computational requirements to the standard error back-propagation algorithm are almost negligible. Theoretical justi"cations and simulation results are given to verify the e!ectiveness of the proposed learning algorithms. ( 2000 Elsevier Science B.V. All rights reserved.